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CASE STUDY 3 


Using Integrated Data Systems to Strengthen 
Collective Impact in Out-of-School Programs 


Solving complex social problems inevitably requires 
action by multiple actors and organizations. In the 
education sector, problems such as chronic absenteeism, 
poor performance in school and low graduation rates are 
increasingly viewed as challenges that schools cannot solve 
on their own. Across the country, collaborative efforts 
are bringing together a variety of organizations that work 
with school-age children and youth, including out-of 
school time programs and nonprofit service agencies, to 


improve educational outcomes in their communities. 


In theory, a shared measurement system that helps partner 
organizations set a common agenda and goals, align activities 
and monitor progress is an essential feature of these collective 
impact efforts. In practice, however, data-informed decision 
making is often stymied because the data needed to 

set goals and monitor progress are collected by separate 
agencies that don’t share information with one another. 


Linking activities to outcomes can be a particular problem 
for nonprofit organizations in collective impact efforts in 
the education sector because these organizations typically 
don't have access to their enrollees’ school records. Nor 
can they easily compare the achievement of their enrollees 
against peers. Without that information, it is hard for 
nonprofit groups to know if their efforts to support 
student performance are effective, how the programs 
might be improved and whether they are allocating 
resources efficiently. 


An integrated data system (IDS) that links the data the 
nonprofit organizations collect on their enrollees with data 
on educational performance collected by the local school 


system can overcome these institutional silos.' 


This case study discusses how two collective impact 
initiatives in the education sector are using data from 
local integrated data systems to guide action, set common 
goals, monitor performance, make programmatic 
adjustments and allocate resources effectively. Both 
initiatives are using an IDS developed and maintained by 
local data intermediaries that are partners in the National 
Neighborhood Indicators Partnership (NNIP).? 


¢ Responding to IDS data that highlighted the poor 
attendance and school performance of its enrollees, 
a collaborative of youth-serving agencies funded by 
the United Way of Central Carolinas took action to 
improve attendance and reduce chronic absenteeism 
as an intermediate step to reach its long-term goal of 
increasing high school graduation rates in Charlotte- 
Mecklenburg County, North Carolina. 


¢ When a consortium of out-of-school time providers in 
the Eastside Promise Neighborhood in San Antonio, 
Texas, saw IDS data that showed their enrollees 
were not improving their proficiency in reading and 
math, the organizations revamped their programs to 
focus on these two subjects. And the local United 


An integrated data system (IDS) periodically links individual-level administrative data from multiple public service agencies and 


contracted service providers, creating a rich picture of individual service needs, participation and outcomes over many years. In 


some systems, individual records are linked together to form comprehensive, longitudinal household and family records. An IDS can 


be operated at the state, county or city level within government or by nonprofit or university partners. By offering large sample sizes, 


longitudinal data and the ability to identify multisystem clients, integrated data systems are valuable tools for policy analysis, program 


planning and monitoring and evaluation. Due to the confidential and sensitive nature of the data, organizations that house an IDS 


carefully follow privacy laws, securely store data and maintain rigorous standards for use and access. For additional information, visit 


www.aecf.org/IDS, www.aisp.upenn.edu and www.neighborhoodindicators.org/resources-integrated-data-systems-ids. 


Way repurposed some in-school programming at the 
elementary level to middle schools and high schools, 
where the in-school supports were more effective. 


MAKING MID-COURSE IMPROVEMENTS: 
CHARLOTTE-MECKLENBURG COUNTY, NORTH CAROLINA 


The United Way of Central Carolinas (UWCC) adopted 
a collective impact model in September 2011, when it 
launched a 10-year initiative to increase the graduation 
rate among at-risk children in the Charlotte-Mecklenburg 
Schools. (A second collective impact effort, focused on 
reducing homelessness, began in 2013.) The collaborative 
includes 16 nonprofit human service and health- 
related organizations funded by the United Way. These 
organizations work with about 16,000 youngsters per 
year. UWCC commissioned the UNC Charlotte Urban 
Institute (UNCC UJ) to track the initiative’s progress by 
matching participant data from the agencies to school 
data in the Institute for Social Capital (ISC) Community 
Database. When the United Way agencies saw the results 
in 2015, they zeroed in on attendance and chronic 


absence as a focus for their common work. 


IDS data revealed a link between chronic absence and poor 
school performance 

The UNCC UI researchers use the ISC Community 
Database to match person-level data collected by 14 
UWCC-funded programs that work directly with school- 


aged children with Charlotte-Mecklenburg school records. 


UWCC and its grantees get two reports: one that shows 

how all the enrollees are doing on various educational 
indicators and one that shows how the enrollees in specific 
agencies are doing. (The agencies see only composite data, 
not information about individual students.) 


The annual reports developed by UNCC UI provide 
aggregated information about participants’ demographics, 
attendance, chronic absenteeism, suspensions, performance 
on standardized tests and graduation rates. Data on the 
12,040 unique program participants enrolled in United 
Way agencies in 2012-2013 (data which became available 
in 2015) showed that the enrollees include a higher 


The Institute for Social Capital (ISC) Community Database, 
the only IDS in North Carolina, includes data gathered 

by governmental agencies (the Charlotte-Mecklenburg 
Schools, Mecklenburg County Department of Social Services, 
Mecklenburg County Sheriff's Office and the Homeless 
Management Information System) and more than 40 nonprofit 
agencies focused on housing and youth services in the 
Charlotte region. Established in 2004, the ISC Community 
Database has been administered by the UNC Charlotte Urban 
Institute since 2012. This database can link individual data 
across agencies, but all released data are de-identified and 
aggregated. The database supports university-based research 
and data-informed decision making in the community. For more 


information, visit https://ui.uncc.edu/programs/isc. 
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proportion of non-white and economically disadvantaged 
students than the school population as a whole. Students 
who spent three or more years in the UWCC-funded 
programs did better on most measures than those who 
participated for only one to two years. But overall, the 
enrollees did worse than other students on key measures: 
‘They were more likely to be chronically absent from school 
(defined as missing 18 or more school days per year), 
and those enrollees who were chronically absent were less 
proficient in reading and math than other students.® 


‘These findings were not surprising, but they were catalytic. 
When UWCC agencies reviewed the data, they focused 
on chronic absence and its relation to school performance. 
Victoria Manning, the former director of Community 
Investment at the United Way, explains, “We saw that the 
children who were served by the organizations we funded 
had strong attendance challenges, and this affected their 
school performance. So we moved forward to focus on 
attendance and chronic absenteeism.” 


Not all organizations funded by UWCC focus on 
academics — the collaborative includes Boy Scouts and 
Girl Scouts as well as Communities in Schools — but they 
can all have an impact on their enrollees’ school attendance, 
Manning notes. Given the strong connection between 
chronic absence and school performance — established by 
national data and confirmed by local data — the UWCC 
agencies felt that doing something about attendance could 
begin to make an impact on school performance. 


Addressing poor attendance by improving services and 
increasing public awareness 
Mobilized by the data, the United Way staff put together an 


action team to develop a community awareness campaign 
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- Susan Hansell, Executive Director, A Child’s Place 


on absenteeism and to focus the member agencies 
on improving attendance among their enrollees. The 
Charlotte-Mecklenburg school system was an immediate 
target for action and a willing partner in the work. 
Participants in the United Way programs attend a total 
of 170 schools in a school district that serves 145,000 
students per year. 


Early on, the school superintendent joined a meeting 
of the workgroup. Emphasizing the importance of 
addressing attendance and chronic absenteeism as a way to 
support the district’s efforts to increase reading and math 
proficiency, the superintendent encouraged the group to 
reach out to the significant adults in children’s lives, not 
just the children. Manning made a presentation on behalf 
of the workgroup to the School Board, the attendance 
problem was discussed at district-wide meetings of school 
principals and the staff person responsible for attendance 
throughout the district quickly became a “champion” of 


the initiative’s work on attendance. 


The United Way agencies and the Charlotte-Mecklenburg 
school system worked together to inform students and 
their parents and caregivers about the importance of 
good attendance. As part of an “attendance awareness 
month” campaign in September 2015, the United Way 
provided all the schools in the district with a one-page 
flyer emphasizing the importance of good attendance and 
reminding parents and caregivers not to schedule doctor's 
appointments or other activities during school hours. 
The schools sent copies of the flyer home with all their 
students. The United Way also sponsored a poster contest 
to promote attendance and awarded prizes and honored 
students for the best posters at the elementary-, middle- 


and high-school levels. 


As UWCC-funded organizations worked to boost school 
attendance among their enrollees, homeless children 
emerged as a particular concern. A 2015 report showed 
that participants who were identified as homeless had 
especially high rates of chronic absence and lower 
academic performance than other students. “It was 
stunning to see how poorly our students were doing 
compared to other children,” says Susan Hansell, executive 
director of A Child’s Place, a Charlotte organization that 
works with about 2,700 homeless children per year. 


“Before getting the IDS data, we focused on school 
promotions and thought we were doing well,” says Hansell. 
“But the IDS data showed that, even though our students 
were being promoted, less than 20 percent were proficient 
in reading and math. And their attendance lagged far 
behind other children. That helped us to see the soft spots 
and have conversations about where we could improve.” 
Because of the data and the discussions they stimulated, 
tracking participants’ school attendance and researching 
and implementing best practices to increase attendance are 
now critical parts of the work of A Child’s Place. Avoiding 
suspensions is another emerging focus. “With the IDS data, 
we could see that although the suspension rate was similar, 
our participants were suspended for a much longer number 
of days than other students,” Hansell says. “Now we are 
working to improve the children’s behavior and emotional 
well-being in order to reduce the number of days they are 
suspended. That will also help to increase the number of 


days they attend school.” 


IDS data galvanized collective action 

“Without the IDS data, we wouldn't have worked on 
attendance as a collaborative,” says Manning. “We knew 
from national data that there is link between attendance, 
chronic absenteeism and school performance. But having 
the local data show that connection made a big difference. 
By seeing what was happening to the children we serve, 


we could bring people together to talk about what to do 
and encourage organizations to pay more attention to 
attendance and absenteeism. We know we won't solve 
everything by focusing on attendance, but we also know 
that if we can address attendance, we can support our 


other academic goals.” 


It is too early for the United Way agencies to see the results 
of their public awareness campaign and organizational 
efforts to improve student attendance, but UNCC UI will 
track these outcomes in the IDS over the course of the 
initiative. In the meantime, the collaborative continues to 
emphasize chronic absenteeism, build new partnerships, 
explore additional strategies and consider related areas of 
work (such as the relationship between suspensions and 
chronic absence) that can heighten its collective impact. 


MONITORING AND MANAGING 
A COLLECTIVE IMPACT EFFORT: 
EASTSIDE PROMISE NEIGHBORHOOD IN SAN ANTONIO 


In the Eastside Promise Neighborhood Initiative, a broad 
consortium of partners works to improve outcomes for 
6,000 children in a five-year cradle-to-career initiative 
in a San Antonio neighborhood of 3.5 square miles 
(www.eastsidepromise.org). In 2012, Eastside Promise 
became one of the first five federally designated Promise 
Neighborhoods in the United States. The United Way 
of San Antonio and Bexar County, which oversees the 
implementation of Eastside Promise, commissioned its 
data partner, Community Information Now (CI:Now), 

to develop an IDS that could be used in Promise 
Neighborhoods and that would eventually be expanded to 
serve organizations, agencies and policymakers throughout 
the city. Partners in the Eastside Promise Neighborhood 
Initiative have used linked data in the CommunityViewer 
IDS to improve service delivery in out-of-school and in- 
school programs. 
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attendance as a collaborative.” 


- Victoria Manning, formerly with the United Way of Central Carolinas 


Initial IDS data showed little improvement 

in enrollees’ school performance 

ClI:Now matches information on the participants 

in out-of-school programs collected by nonprofit 
providers to education data collected by the San Antonio 
Independent School District. The United Way accesses 
CommunityViewer’s summary results and shares the data 
with the individual agencies. Providers see summary data 
on the participants’ standardized test results, grades and 
school behavior, including disciplinary actions, suspensions 
and attendance. (Ifa provider wants to discuss the progress 
of individual students, the United Way arranges for school 
officials to be involved in that conversation.) 


To understand the results and use them in decision 
making, the United Way formed a community of practice 
with the service providers that operate out-of-school time 
(OST) programs in Eastside Promise. Representatives 
(typically, the executive director) of 8-10 organizations 
met twice a month for about seven months in 2014. 

In calendar year 2014, OST programs funded by the 
United Way served between 500 and 800 students in each 
school semester and during the summer. Examining the 
data from the IDS, organizations in the group discussed 
ways of targeting their resources and improving program 
offerings to have the most impact on the greatest number 
of students. “Having active partners who think through 
what the data mean and their implications for practice 

is a critical element in using data to improve collective 
impact,” says Henrietta Mufioz, vice president for 
Grant Research and Evaluation at the United Way. Each 
organization in the community of practice agreed to 


share its enrollees’ school results with the other partners. 
The shared data and open dialogue helped partners work 
effectively as a group to identify and strengthen weak spots 
in their programs, with the goal of serving students better. 


Cl:Now worked closely with United Way and the service 
providers to analyze and stratify the data, enabling the 
providers to develop a fuller understanding of their 
enrollees and their school performance. Providers could see 
how well they were doing with certain types of students 
— for example, those who started out as lower performers 
— and determine how well their enrollees were doing 
compared with other students in the same schools. 


The community of practice learned that while the overall 
pass rate on standardized reading and math tests was 
going up in local schools, enrollees in OST programs were 
not improving their scores. These data led the group to 
focus on reading and math proficiency and improvements 
in English and math grades as key indicators of progress 
and to treat reading and math help as critical elements in 
their after-school programs. 


Reshaping programs and allocating resources more effectively 
Out-of-school time programming. Once the direction 
and targets were clear, one faith-based organization 
modified its after-school curriculum to help its enrollees 
boost their reading and math scores. While it had 
previously offered mentoring and tutoring in a variety 
of subjects, the organization refocused its program to 
concentrate on English and math and increased the 


amount of time enrollees spend on those two subjects. 


CommunityViewer is an IDS developed and managed by Community Information Now (CI:Now), a data intermediary in the San Antonio 


region. Established in fall 2012, the CommunityViewer database links together individual-level information from multiple private data 


sets. As of 2016, CommunityViewer contained data from the San Antonio Independent School District and data collected by 15 local 


service organizations. Currently, the primary users are the partner organizations operating the federally designated Promise and Choice 


Neighborhoods initiatives in San Antonio. CommunityViewer is supported by the United Way of San Antonio and Bexar County. For more 


information, visit http://cinow.info/communityviewer. 


The United Way continues to use IDS data on enrollees’ 
school performance to monitor the providers’ collective 
and organizational performance. Annual reviews of the 
data show that the OST program students are making 
small, durable improvements. “Program participants have 
made modest gains and maintained slightly improved 
grades in English and math,” says Mufioz. “We would 
be hard pressed to argue that they will move from a C to 
an A, but they can move from a D to aC, and they can 
maintain the gains they make. Considering where they 
started, that’s a victory.” 


In-school programming. United Way also used data from 
the IDS to make decisions about the most effective use 
of resources for in-school programs in Eastside Promise. 
Impressed by an evidence-based in-school program using 
peer mentors and other classroom supports in middle 
schools and high schools, the United Way asked the 
organization to operate the program in two elementary 
schools. But when the participant data was matched to 
school records data in the IDS, CI:Now found that this 
program model was not effective at the elementary-school 
level. Instead of showing educational gains, enrollees 
were showing declines. Concluding that more specialized 
help was needed for younger children, the United Way 
stopped funding the program in the elementary schools. 
It redirected the money to operate the program in more 
middle schools and high schools (where it was effective) 
and tried a different approach in the elementary schools. 


IDS increases impact of service providers, 

United Way and Eastside Promise 

Mufoz points out that the IDS data on enrollees’ school 
performance is information the out-of-school providers 
never had before. Previously, they had only self-reported 
information from the participants — information that 
is inevitably incomplete or inaccurate. For the first 

time, providers can see an accurate picture of how their 
enrollees are doing in school, how they compare with 
other students in the schools and how they compare with 
enrollees in other OST programs. Having these data is 
equally eye opening for the United Way, which funds 

a total of 80 organizations and 120 different programs. 
In the past, the United Way had to measure success in 
terms of activities and outputs — such as, the number 
of participants served and the services they received. 
Now, the United Way can see how the participants are 
faring on a variety of educational outcomes and track 
their progress over time. “Having the IDS data allows 

us to understand what it means to have impact in the 
community,” says Mufoz. “We can tell whether the 
programs we fund are making a difference. This is taking 
the United Way organization into the next generation 
of accountability for impact.” CI:Now Director Laura 
McKieran agrees. The IDS allows Eastside Promise and 
its partner agencies to make “the most effective use 

of resources among the population the programs are 
intended to serve,” she says. 


eee Yaving the [DS dataalowsusto Ze 
understand what it means to have 

impact in the community. We can 

tell whether the programs we fund 

are making a difference.” 


- Henrietta Mufioz, United Way of 


San Antonio and Bexar County 
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Endnotes 


| This case study is one of a series that shows how state 
and local policymakers and practitioners use IDS data 
to improve policies, programs and practice. For more 
information, visit www.aecf.org/IDS 


Mm 


The National Neighborhood Indicators Partnership 
(NNIP) comprises independent data intermediaries in 
30 cities that have a shared mission to help community 
stakeholders use data for better decision making. Several 
NNIP partners, in addition to those in Charlotte and 
San Antonio, maintain or work with an IDS. For more 
information, visit www.neighborhoodindicators.org 
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Nineteen percent of the United Way enrollees were 
chronically absent, compared to 10 percent of all students 
in the Charlotte-Mecklenburg Schools. Only 14 percent 
of the chronically absent United Way enrollees were 
proficient in reading, and only 9 percent were proficient 
in math. In contrast, among all CMS students, proficiency 
rates were 27 percent in reading and 25 percent in math. 
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